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EVALUATION OF THE 1981 BASIC SKILLS SUMMER SCHOOL PROGRAM: 
AN ASSESSMENT OF PROGRAM IMPACT 

) EXECUTIVE SUMMARY \ 

BACKGROUND 

The Board of Education of Montgomery County Public Schools adopted a 
resolution in May, 1980, establishing a Basic Skills Summer School Program. 
The purpose of the program was to provide intensive remedial instruction in 
the basic skills areas of reading, writing, and mathematics to pupils who 
scored very low on nationally standardized achievement tests. 

The present report presents data which indicate the short-term effects of the 
1981 Basic Skills Program on Reading and 'Mathematics and a follow-up 
assessment of the program's impact on pupil performance during the regular 
school year. 

FINDINGS 

QUESTION 1: Did pupils who participated in the Basic Skills Summer School 

Program make significantly hig h er gains from pretest to posttest In Readi ng, 
Vocabulary, and Math ematics than pupils who did not participat e in the 
program? 

Overall, the results from analysis of test scores were mixed. 
Participants scored significantly higher (p<.05) than nonpart ici pants in 
^ Mathematics overall and in Grades 7 and 8. There were no significant 
differences in test scores of participants fnd \onparticipants in Reading 
or Vocabulary. However, both overall and at each grade level the 
observed differences in Reading favored participants, even though' the 
results were not statistically significant. 

Question 2: Did participants in the Basic Skills Summer Program perform 

better than nonparticlpants during the first semester of the regular school 
year in terms of grad es and/or work study habits in Engli sh/Language Arts 
Mathematics, and Reading? — s ' 

q Overall, at the end of the first semester the grades of program 
participants in all areas were higher or showed greater improvement than 
those of nonparticlpants. Participants also received higher marks and 
showed greater improvement in work-study habits than nonparticlpants. 

Question 3: Did participants in the Basic Skills Summer School Program * 

exhi bit more positive school related behaviors than nonparticlpants in term s 
of better attendance, fewer tardles, and/@r fewer times and days suspended? 

Overall, results from analysis of school-related behaviors showed that' 
school attendance of former participants was significantly higher (p<.05) 
than that of the nonparticlpants. There were no differences in the 
average number of tardies and suspensions of the two groups. 
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CONCLUSIONS 



While effects are modes^-^e^^kfd ings from this evaluation of the 1981 Basic 
Skills Summer School Program sugg^t^that the program may be succeeding in 
reaching its goals. Specif icallyj analyses of the short-term effects of the 
program suggest that while the changes for Functional Reading were not 
sufficiently large to be of sta/tistical significance, they were in the 
positive direction, with participants outperforming nonpar t icipant s . 
mathematics, however, the data show that the program was effective 
improving pupil scores significantly, particularly for the upper grades. 



In 
in 



follow-up study show positive long-term 



0 



am 
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The findings from the 

effects. Data collected at the end of the first semester show that there is _ 
consistent trend in each of the areas examined showfng that participants In 
the Basic Skills Summer Program outperformed their nonparticipating peers in 
their regular school program. . This finding is extremely encouraging, as 
lasting' results are not typically found for programs of such-shoA duration. 
Further, it is consistent with the findings from the test data reported 
earlier that smaller differences are found in content knowledge, i.e., GPA, 
than in what might be considered general behavioral areas, i.e., work study 
skills and attendance. * 

It is not entirely possible to rule out "self-selection" in explaining these 
differences, as the two groups differed in a very important- variable, their 
wilLingness to attend the summer school program. Nonetheless, it seems 
reasonable to be cautiously optimistic about these study findings and to 
conclude that the program appears to be having a positive impact on 
part ici pants . 
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EVALUATION OF THE 1981 BASIC SKILLS SUMMER SCHOOL PROGRAM : 
AN ASSESSMENT OF PROGRAM IMPACT 



BACKGROUND 




The Board of Education of Montgomery County Public Schools adopted a 
resolution In May, 1980, establishing a Basic Skills Summer School Program. 
The purpose of the pf-Q£r4m was to provide Intensive remedial Instruction in 
the basic skills areas o£~pfeading, writing, and mathematics to pupils who 
scored very low on nationally standardized achievement tests. 

The Department o f " Edu£a~t ional Accountability was requested to conduct an 
evaluation of the 1981 program (second year) to obtain data regarding program 
Implementation and effectiveness. 

The present report, the final for this year's evaluation, presents data which 
indicate the short-term effects of the Basic Skills Program in Reading and 
(athematlcs and a follow-up assessment of the program's impact on pupil 
performance during the regular school year. Specifically, it is designed to 
answer the following questions: ' 

I. What are the short-term effects of the Basic Skill-s Summer School 
Program on pupil achievement in Reading and Mathematics? 

v 2. What are the longer-terra effects of the Basic Skills School Program 
as measured by pupil grades, work study habits, and behavior during 
the first semester of the regular school year (1981-82)? 

rn Interpreting the findings of this evaluation, two major considerations 
nust, however, be kept in mind. 

I. Pupils in -both the treatment and control group were administered 
posttests after having been in the regular school program for almost 
two months. This instructional period exceeds the length of the 
Basic Skills Program; therefore, any differences between the two 
groups at the immediate end of the program might possibly have been 
reduced or expanded by instruction. 

Although every attempt was made to use test instruments matching the 
content of - the summer program instruction, the extent to which this 
goal was achieved Is unknown. 

\ summary of the procedures used to make these assessments ,1s shown In 
Exhlbl t 1 . 
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EXHIBIT i 

Procedures used for Assessment of Basic Skills 
Summer School Program Impact on Pupil Achievement and Behavior 



Asse S8men t 
Component 



Short-term Assessment 



Sample 



I nstrumentatlon 



The original sample included 1,157 
pupils in Grades 4-8. However, 
due to student absentees, pupil 
transfers, etc., during pre/post- 
testing, the final sample con- 
sisted of 774 pupils (326 partici- 
pants and 448 nonpartlclpants) . 



o MFRT: Form A 

o MFRT: Form B 

o MCPS Basic 
Skills Reading 
Test 

o Objectives 
Referenced 0 
Basic Skills 
Mathemat lcs 
Test 



Data Sources 



Test scores resulting 
from pretests and post- 
tests which were admin- 
istered to all eligible 
pulls June 1981 and 
October-November I 981 , 
respectively. 



D ata Analysis 

Posttest scores of 
participants and non- 
participants were 
analyzed using 
analysis of covar.i- 
ance with pretest's 
and grade level as 
the covarlate. 



Follow-up Assessment 



The sample Included 221 pupils 
(114 participants and 107 non- 
participants) enrolled In grades 
7-8 who attended the same school 
the year before and after the 
summer school program. 



o Stanford Diag- 
nostic Auditory 
Vocabulary Test: 
Green Le.vel 



qP"ryland Functional Reading Test 



Pupil grade point 
averages , workstudy 
habit* average, atten- 
dance, and suspension 
data were collected 
from report cards for 
the first semester 
(1981 and June 1 981) . 
The subjects areas 
reviewed were Reading, 
Mathemat lcs , and 
^ngl 1 sh/fcanguage Ar t s . 



Analysis of covarl- 
ance was used to com- 
pare the first semes- 
ter 1982 grades point 
averages, work study 
habit averages, and 
attendance of partic- 
ipants and nonpar- 
tlclpants. <5^2$de 
point average, work 
study habits, and,< # 
attendance from I^#U 
were used as th*» co- 
varlates. T-tests 
were used to analyze 
suspension rates. 
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FINDINGS 



One of the primary questions which this section of the evaluation Is designed 
to answer Is, "Did pupils who participated In the Basic Skills Summer School 
program make slgnl ficantly # higher gains from pretest to posttest In Reading 
Vocabulary, and Mathematics than pupils who did not participate In the 
program?" , 

Overall, the results from analysis of the tests administered before and after 
the Basic Skills Summer School Program were mixed. (See Appendix A-l for 
details.) Specifically, those data showed 

\ 

o There were no significant differences between the posttest scores of 
participant and nonpa r t ic L pan ts in Reading. However, both overall 
and at each grade, differences favored participants. 

o There were no significant differences between the posttest scores of 
participants and nonpart ici pants in Vocabulary. 



Participants in the Basic Skills Summer School Program scored 
significantly higher Cp<<.05) than nonpar t ic 1 pant s In Mathematics 
overall and in Grades 7 and 8. The differences between parti- 
cipants and nonparticipants in Grades 4, 5/and 6 were not signifi- 
cant . 



Analysis of pre/posttest scores within each of the seven summer school centers 
also produced mixed results. The differences detected between participants 
md nonparticipants in most instances were not significant; however, in four 
of the centers, significant differences were obtained (see Appendices A-2 -and 
\-3 for details). Specifically those data showed: 

o eighth grade participants who attended the Newport Center scored 
significantly (p<.05) higher on the MFRT: Form B than 8th grade 
nonparticipants. 



Sixth grade participants 
significantly (p .05) 
nonpa rt ic Ipant s . 



who at tended the Lee 
lower on the _ MFRT: 



Cen te r 
Form 



scored 
A than 



Overall participants who attended the Julius West Center scored 
significantly (p<.05) higher in Mathematics than nonparticipants. 

Eighth grade participants who attended the Takoraa Park Center scored 
significantly (p<.05) higher in mathematics than nonparticipants. 
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The next question posed in this investigation is, "What are\ the longer-term 

effects of the Bas~£c Skills School Program as measured by pupil grades, work 

study habits, and behavior during the first semester of the J regular school, 
year (1981-82)? ~J 

To address this question, the following issues were exarairre^ 

1. Did participants in the Basic Skills Summer Program perform better 
than nonparticipants during the first semester of the regular school 
year in terms of achievement and/or work study habits in English/LA, 
Mathematics, and Reading? ' . 

2. Did participants in the Basic Skills Summer School Program exhibit 
more positive school-related behaviors than nonparticipants in terras 
of" better attendance, fewer tardies, and/or fewer times and days 
sus pended? * * 

/ 

The results of this investigation are summarized in the following series of 
exhibits, which show slight, but definite, positive program effects . The 
discussion of program effects which follows the graphs are niade with reference 
to both the graphs and the results of analysis of covariance, the latter of 
which are shown in Appendix A-4 . 

v- 

GRADES ' a 

Overall. ~ at the end of the fir'st semester, the grades of program participants 
were higher or showed greater improvement than those of nonparticipants (see 
Exhibit 2). Specifically, 

o Participants in the summer school program had significantly (p.<.10) 
higher grade point averages than nonparticipants in Errglish/LA. 

.o Participants and nonpartic ipants^ had identical grade point averages 
in Mathematics. However, in the previous academic year, grade point 
averages in Mathematics for participants had* been significantly 
( p < . 05 ) lower than that of non pa rtici pants. 

o Participants and nonparticipants did not differ significantly in 
\ ^--trfieir grade point averages in Reading. 



Typically a difference is not considered statistically significant unless 
it is attained at the .05 level or significance or less. Tn the present 
exploratory analyses, however , we have relaxed this criterion and considered a 
difference which reaches the .10 level to be significant. This is a common 
practice where a study is exploratory in nature, when the sample sizes are 
fairly small, and when trends are considered to be impor&ant. 4 




ENGLISH/LA 




June 1981 
?Pre) 



Jan 1982 
(Post) 



Grade Point 
Average 



READING 



3.0 



2.5 



2.0 4 



N=27 



N=29 



June 1981 *' 
*(Pre> 



T 



Jan 1982 
"(Post) 



WORK STUDY HABITS 



Overall, at the end of the firs£ semester, work study habit marks of program 
participants were higher or showed greater improvement than those of 
nonparticipants (see Exhibit 3). Specifically, 

o Participants had significantly higher work study habit averages than 
nonparticipants in English/Language Arts (p <C .09) and Reading 
(p«.01). 

^ o Participants and nonparticipants did not differ significantly in 
their marks in work^study habits iri mathematics at the end of the 
first semester. However, in the previous academic year, 
participants 1 work study habit averages had been significantly 
(p<^.05) lower for participants than nonparticipants. 
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EXHIBIT 3 



Pupil Work Strafly Halbitts By PauptScipatioia {Pre/Post) 



ENGLISH/LA 



MATHEMATICS 



Work Study 
Habits 



1.0 



0.5 4 



N=66 




Work Study 
Habits 



1.0 



0.5 



N<0>IQ]pGlI?ttIci]p)SlMt© 




June 1981 
(Pre) 



Jan 1982 
(Post) 



Junes 1981 
(Pre) 



Jan 1982 
(Post) 
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Work Study 
Habits 



1.5 



1.0 - 



0.5 4 



I 



READING 



N=21 




June 1981 
(Pre) 



Jan 1982 
(Post) 



1 o 



BEHAVIOR 
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Overall, results from analysis of school-related behaviors were mixed. (See 
Exhibit 4.) Specifically, 

o School attendance by^participants for the first semester (1982) was 
significantly (p<.05) higher than that by nonparticipants . 

o <v Differences in the average number of2tardies between participants 
N^nd nonparticipants were not significant. 

o Differences in the average number of suspensions as well as the 
average number of days suspended between participants and 
nonparticipants were not significant. 



2 

Data on tardies and suspensions are not presented in graphic form because 
only current year data were collected. 



EXHIBIT 



Puapal Atteimdajace By Psurtidpsitsoisi (Pre/Post) 



ATTENDANCE 



Average Days 
Attsadancs 



85.0 -=> 



84,0 



S3.0 J 



NrajpsiFticiipaiiats 



N=97 



June 1981 
(Pre) 



N=93 



Jan 1982 
(Post) 
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While effects are modest, the findings from this evaluation of the 1981 Basic 

Skills Summer School Program suggest that the program may be succeeding in 

reaching its goals. Specifically, analyses o£ the short-term effects of the 

program suggest that while the changes for 1 Functional Reading were not 

sufficiently large to ba of statistical significance, they were in the 

positive direction, with participants outperforming nonpar ticipants . In 

mathematics, however, the data show that the program was effective in 
improving pupil scores significantly, particularly for the upper grades. 

The findings from the follow-up study show positive long-term program 
effects. Data collected at the end of the first semester show that there is a 
modest, but consistent trend in each of the areas examined' showing that 
participants in ^he Basic Skills Summer Program outperformed their 
nonparticipating peers in their regular school program. This finding is 
extremely encouraging, as lasting results are not typically found for programs 
of such-short duration. Further, it is consistent with the findings from the 
test data reported earlier that smaller differences are found in content 
knowledge, i.e., GPA, than in what might be considered general behavioral 
areas, i.e., work study skills and attendance. 

Tt is not entirely possible to rule out "self-selection" in explaining these 
differences, as the two groups differed in a very important variable, their 
wiLLlngness to attend the summer school program. Nonetheless, it seems 
reasonable to be cautiously optimistic about these study findings and to 
conclude that the program does appear to be having a positive impact on 
participants. 
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APPENDICES 



APPENDTX A-1 



0 



Adjusted Posttest Means ln Reading and Mathematics by 
Participants and Nonpart lclpants 















MATHEMATICS 






VOCABULARY 




G rade 


Part 


Nonpart 


Dlf f 




Part 


Nonpart 


Dlff ■ 


Part 


Nonpart 


Diff 


4th 


54.48 
{Z/ ) 


53.95 
(15) 


+0. 53 


NS 


9.71 
(29) 


12.45 
(17) 


-2.74 NS 


30.59 
(29) 


32.41 
(17) 


-1 . 82 NS 


5th 


65.83 
CI j) 


64.25 
(18) 


+1 . 58 


NS 


13.51 
(15) 


14. 52 
(18) 


-1 .01 NS 


34.38 
(15) 


34.70 
(19) 


-0.32 NS 


6th a 


91.34 
(121) 


90.86 
(184) 


+0.48 


NS 


18.16 
(115) 


17.48 
(165) 


+0.68 NS 








7th 


7 5.22. 
(135) 


74.96 
(179) 


+0.26 


NS 


21.16 
(132) 


19.60 
(176) 


+1 . 56*** 








8th b 


105.8 
(21) 


103.39 
(40) 


+2.42 


NS 


24.05 
(20) 


21.52 
(35) 


+2. 53* 








Total c 


72.03 
(179) 


71 .63 
(213) 


40.40 


NS 


19.10 
(311) 


18.17 
(411) 


+0. 93** 


32.23 
(45) 


33.36 
(36) 


-1.13 



a 

Seventh grade Maryland Functional Reading Test: Form A 
Ninth grade Maryland Functional Reading Test: Form R 

C MCPS Basic Skills Reading Test 

NS Not significant p .05 



.05 
.01 



***p .001 
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APPENDIX A-2 



Analysis Of Adjusted Posttest Means By Participants and Nonpar t ic ipants Within Summer School Centers 

(Grades 6-8) 



NEWPORT 



Grade 



READING 
6th 



7.th 



8th, 



Part 


-Part 


(N) 


(N) 


93. 34 


90. 71 


(29) 


(45) 


72.56 


80.89 


(13) 


(12) 


117.12 


109.1 


(6) 


(10) 



Diff 



WEST 
Part 
(N) 



-Part 
(N) 



Diff 



LEE 

Part 

(N) 



-Part 

00 



Di 



RIDGE VIEW 
Part -Part 
(N) (M) 



Diff 



TAKOMA 
Part -Part 



(N) 



(N) 



Diff i 



+2 


.63 


87.49 


87.66 


-0 


17 


94.92 


100.08 






U5) 


(37) 






(33) 


(47) 


-8 


.33 


78.82 


76.91 


+1 , 


31 


75.13 


75.00 






(27) 


(22) 






(57) 


(63) 


•f7 


.98* 










99.25 


104.76 














92) 


(6) 



-5 


.16* 




P.A Qfi 




no 


89.39 


87 .75 


+1 


.64 






(9) 


(29) 






(35) 


(32 ) 






. 1 3 


AS 7R 
D J • / O 




+ 1 


. 23 


79. 96 


78.31 


+1 


.65 






(il) 


(37) 






(28) 


(46) 




j 












101 . 1 1 


101 .02 




.09 














(13) 


(24) 






-0 


28 


1 8 . 30 


13.78 


-0 


48 


15.02 


-J4*44 


M) 


.58 






(9) 


(26) 






(37) 








-K). 


88 


18. 77 


16.57 


+2. 


20 


19.24 


18.32 


+0 


92 






U2) 


(42) 






(25) 


(44) 


+1 . 


68 










25. 72 


22.11 


+3. 


6i* 














(11) 


(18) 






-K). 


50 


16. 59 


15.45 


+1 . 


14 


18. 78 


17.37 


+1 . 


41* 






(21) 


(68) . 






(73) 


(88) 





V 

MATHEMATICS 
6th 19.40 
(27) 

7th 23.72 
(U) 

8th 20.80 
(6) 

Total 20.58 
' (440 



18.60 
(43) 

21 . 57 
(9) 

19.42 
(10) 

19.23 
(62) 




-K).80 



+2 .15, 



+1 . 38 
+1 . 35 



22 . 19 
(37) 



17.58 


+2 


.19 


21 . 22 


21.50 


(36) 






• (31) 


(34) 


23.15 


+1 


.66 


21 .23 


20. 35 


(20) 






(58) 


(61) 








24. 58 


22.90 








(3) 


(7) 


20. 30 


+1 . 


89* 


21 .37 


20.87 


(56) 






(92) 


(102) 



Maryland Functional Reading Test: Form A 
Maryland Functional Reading Test: Form B 
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APPENDIX A-3 

Analysis Of Ad justed. Posttest Means By Grade Within Summer School Centers 

(GRADES 4-5) 



Grade 


T 1 

0 

STRATHMORE 
Part — Part 
(N) (N) 


I'll L 


WELLER ROAD 
Part -Part 
(N) (N) 


Dlf f 


VOCABULARY 
4th 


3^.11 32.09 
(5) (5) 

r 


+1.02 
(24) 


30 . 17 
(12) 


32 . 32 


-2.15 


5th 


S 


(1 4) 


34 .44 

C*>6) 


34.69 


-0.25 


Total 


33.35 32.98 
(6) (8) 


+0.37 


32.04 
(38) 


33.27 
(25) 


-1.23 


READ IMG 
4th 


32.24 43.56 
(5) (5) 


-11.32 


60 . 14 
(22) 


57 . 79 
(10) 


+2 .35 


5th 


w 




68.54 
(14) 


65.02 
(15) 


+3.52 


Total 


35.25 47.06 
(6) (8) 


-11.81 


64.06 
(36) 


61.03 
(25) 


+3.03 


^tATHEMATICS 
4th 






10. 76 
(24) 


6=01 .66 
(12) 


-0.90 


5th 






14.09 
(14) 


14.84 
(15) 


-0.75 


Total 






12.23 
(38) 


13.07 
(27) 


-0.84 



• / 



•J 
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APPENDIX A-4 



Ad justed" First Semester GPA's, WSH, ^and Attendance 
By Participants and Nonpartieipants 



Participants 



Nonpartieipants 



Variable 






00 


X 


(N) 


Diff . 


Sign. 


English 






















GPA 


1 


.93 


(102) 


1 


74 


(92) 




.19 


P 


.10 


WSH 


0 


.91 


(66) 


0 


68 


(61) 


1 +0 


.23 


P 


.09 

> 


Mathematics 






















CPA 


2 


.13 


(106) 


2 


05 


(96) 


40 


.08 




NS 


WSH 


I 


.01 


(66) 


0 


87 


(63) 


40 


14 




NS 


Reading 






















GPA 


2 


.40 


(27) 
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WSH 2=0uts tanding; l=Sat is factory ; 0=Need Improvement^ 
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